LENS CORRECTION USING PROCESSED YUV DATA 

by 

HoSeong Lee 

TECHNICAL FIELD 

The present invention relates to image sensors, and more particularly, to a lens 
correction method for use on the processed output of an image sensor. 
BACKGROUND 

Integrated circuit technology has revolutionized various fields, including 
computers, control systems, telecommunications, and imaging. In the field of imaging, 
the charge coupled device (CCD) has been made popular by its performance 
characteristics. Nevertheless, the solid state CCD integrated circuits needed for 
imaging are relatively difficult to manufacture, and therefore are expensive. In addition, 
because of the differing processes involved in the manufacture of the CCD integrated 
circuits relative to MOS integrated circuits, the signal processing portion of the imaging 
sensor has typically been located on a separate integrated chip. Thus a CCD imaging 
device includes at least two integrated circuits: one for the CCD sensor and one for the 
signal processing logic. 

Another class of image sensors is the CMOS active pixel sensor. As noted in 
U.S. Patent No. 5,625,210 to Lee et al. ("the '210 patent"), an active pixel sensor refers 
to an electronic image sensor with active devices, such as transistors, that are 



EXPRESS MAIL NO. EV 336042819 US 
Attorney Docket: 38493-8076.US01 



associated with each pixel. The active pixel sensor has the advantage of being able to 
incorporate both signal processing and sensing circuitry within the same integrated 
circuit because of the CMOS manufacturing techniques. 

Because of the advances in image sensor technology, image sensors are now 
commonplace and are used in small form factor applications, such as personal digital 
assistants (PDAs) and cell phones. As the size of the image sensor and associated 
lens decreases, it has been found that increased aberration occurs. Aberration results 
in undesirable effects, such as shading, "dark corners", color differences, 
"reddishness", and vignetting. 

FIG. 1 shows one approach for attempting to correct some of the problems 
mentioned above. FIG. 1 shows lens 102, a sensor array 104 associated with an 
image sensor module (not shown), Bayer RGB data 106, a lens correction process 108, 
an RGB data output 110, image processing 112, and image processed YUV/RGB data 
114. As shown in FIG. 1, the lens correction processing is performed on the raw RBG 
data output 110. Such an approach facilitates easy color control associated with 
control of automatic white balance 1 12a (AWB), color matrix 1 12b, color gain/hue 1 12c, 
color gamma correction 1 1 2d, YUV conversion 1 12e, and other image processing 1 12f. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is illustrated by way of example, and not by way of 
limitation, in the figures of the accompanying drawings and in which like reference 
numerals refer to similar elements and in which: 

FIG. 1 is a schematic diagram that illustrates one approach for lens correction. 

FIG. 2 is a schematic diagram that illustrates a process for lens correction in 
accordance with certain embodiments of the invention. 

FIG. 3 is a schematic diagram illustrating that the lens correction is performed on 
the YUV data in accordance with certain embodiments of the invention. 

FIG. 4 is a diagram that illustrates a set of coordinates used in the correction 
method according to certain embodiments of the invention. 

DETAILED DESCRIPTION 

The present invention is a lens correction method and apparatus which can 
improve, enhance, or solve the shading effects of an image produced by a given lens, 
dark corner effects on the image, color differences between the center and corners of 
the image, and vignetting problems in various types of image sensors. 

FIG. 2 is a schematic diagram that illustrates a process for lens correction in 
accordance with certain embodiments of the invention. In FIG. 2 shows a lens 202, a 
sensor array 204 associated with an image sensor module (not shown), Bayer RGB 
data 206, a lens correction process 208, image processing 212, and image processed 
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YUV/RGB data 214. Image processing 212 comprises controls associated with 
automatic white balance 112a (AWB), color matrix 112b, color gain/hue 112c, color 
gamma correction 112d, YUV conversion 112e, and other image processing 112f. In 
FIG. 2, lens correction process 208 is applied to the image processed YUV data 214. 
As seen in FIG. 2, because certain embodiments of the invention use the image 
processed YUV data, there is no need to implement the lens correction method in the 
image sensor module. Instead, the lens correction method may be implemented on a 
digital signal processor (DSP), a backend processor, or an image processor. The lens 
correction method can be implemented by software on a general purpose central 
processing unit (CPU) or microprocessor or by a combination of software and 
hardware. 

In the following description, numerous specific details are provided to provide a 
thorough understanding of the embodiments of the invention. One skilled in the 
relevant art will recognize, however, that the invention can be practiced without one or 
more of the specific details, or with other methods, components, etc. In other 
instances, well-known structures or operations are not shown or described in detail to 
avoid obscuring aspects of various embodiments of the invention. 

Reference throughout the specification to "one embodiment" or "an embodiment" 
means that a particular feature, structure, or characteristic described in connection with 
the embodiment is included in at least one embodiment of the present invention. Thus, 
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the appearances of the phrases "in one embodiment" or "in an embodiment" in various 
places throughout the specification are not necessarily all referring to the same 
embodiment. Furthermore, the particular features, structures, or characteristics may be 
combined in any suitable manner in one or more embodiments. 

As noted above, certain embodiments of the invention operate in the YUV color 
space (domain). FIG. 3 is a schematic diagram illustrating that the lens correction is 
performed on the YUV data in accordance with certain embodiments of the invention. 
FIG. 3 shows that a Y correction value 204a is applied to the Y data 202a to result in a 
corrected Y data 206a. Similarly, a U correction value 204b is applied to the U data 
202b to result in a corrected U data 206b. A V correction value 204c is applied to the V 
data 202c to result in a corrected V data 206c. 

The equations presented herein are merely illustrative and may vary from 
implementation to implementation. In simple terms, certain embodiments correct each 
component of the image data in the YUV domain in accordance with: 

Corrected Y = Y*YGain Equation 1 

Corrected U = U + DeltaU Equation 2 

Corrected V = V+ Delta V Equation 3 

where, 

YGain is the Y correction value 
DeltaU is the U correction value 
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DeltaV is the V correction value 

For purposes of explanation, any pixel to which a correction value (either Y, U or 
V correction value) is to be applied is referred to herein as the target pixel. According 
to certain embodiments, the Y, U and V correction values are based on the distance of 
the target pixel relative to the center of the reference image. 

According to certain embodiments, the U correction value is based on a first 
distance value. Such a U correction value is bounded by a minimum and a maximum U 
correction limit, which are constants. The minimum and maximum U correction limits 
may be user-selected based on properties of the lens. The user may select such 
minimum and maximum U correction limits so that the resulting U correction value is 
neither insufficient nor excessive when applied to a given target pixel. 

With respect to the first distance value, assume that the pixels in the reference 
image trace a set of imaginary concentric rings emanating from the center of the 
reference image. The same U correction value is applied to each pixel (target pixel) on 
a given ring. In such a case, the distance value may be determined by a function 
defined by Equation 10 below, as an example. 

According to yet another embodiment, the first distance value may be defined by 
assuming that the pixels in the reference image trace a set of imaginary concentric 
rhombuses (diamond shape) emanating from the center of the reference image. The 
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same U correction value is applied to each pixel (target pixel) on a given rhombus. In 
such a case, the distance value may be determined by a function defined by Equation 
1 1 below, as an example. 

According to another embodiment, the first distance value may be defined by 
assuming that the pixels in the reference image trace a set of imaginary concentric 
rectangles emanating from the center of the reference image. The same U correction 
value is applied to each pixel (target pixel) on a given rectangle. In such a case, the 
distance value may be determined by a function defined by Equation 12 below, as an 
example. 

According to another embodiment, the first distance value may be defined by 
assuming that the pixels in the reference image trace a set of imaginary concentric 
polygons that are approximately ring-shaped and emanating from the center of the 
reference image. The same U correction value is applied to each pixel (target pixel) on 
a given polygon. In such a case, the distance value may be determined by a function 
defined by Equations 13a, 13b, 13c and 13d below, for example. Such polygons are 
intended to be approximations of the concentric rings of Equation 10. The polygons of 
Equations 1 3a, 1 3b, 1 3c and 1 3d are twelve-sided. However, the embodiments are not 
restricted to twelve-sided polygons. The greater the number of sides to a given 
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polygon, the closer the approximation to a ring. Thus, the level of approximation to a 
ring can be user-selected. 

Further, the U correction value may either be increased in decreased by a 
luminance parameter that is related to the Y value of the target pixel. Such a 
luminance parameter is user-selected based on whether the Y value of the target pixel 
satisfies certain luminance-related conditions. The luminance-related conditions 
associated with the U correction value may be defined by Equations 8a, 8b, and 8c 
below, for example. 

Thus, based on the above description, the U correction value, namely, DeltaU, 
may be defined by Equation 5 below, as an example, according to certain 
embodiments. 

According to certain embodiments, the V correction value is based on a first 
distance value. Such a V correction value is bounded by a minimum and a maximum V 
correction limit, which are constants. The minimum and maximum V correction limits 
may be user-selected based on properties of the lens. The user may select such 
minimum and maximum V correction limits so that the resulting V correction value is 
neither insufficient nor excessive when applied to a given target pixel. 

With respect to the first distance value, assume that the pixels in the reference 
image trace a set of imaginary concentric rings emanating from the center of the 
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reference image. The same V correction value is applied to each pixel (target pixel) on 
a given ring. In such a case, the distance value may be determined by a function 
defined by Equation 10 below, as an example. 

According to yet another embodiment, the first distance value may be defined by 
assuming that the pixels in the reference image trace a set of imaginary concentric 
rhombuses (diamond shape) emanating from the center of the reference image. The 
same V correction value is applied to each pixel (target pixel) on a given rhombus. In 
such a case, the distance value may be determined by a function defined by Equation 
1 1 below, as an example. 

According to another embodiment, the first distance value may be defined by 
assuming that the pixels in the reference image trace a set of imaginary concentric 
rectangles emanating from the center of the reference image. The same V correction 
value is applied to each pixel (target pixel) on a given rectangle. In such a case, the 
distance value may be determined by a function defined by Equation 12 below, as an 
example. 

According to another embodiment, the first distance value may be defined by 
assuming that the pixels in the reference image trace a set of imaginary concentric 
polygons that are approximately ring-shaped and emanating from the center of the 
reference image. The same V correction value is applied to each pixel (target pixel) on 
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a given polygon. In such a case, the distance value may be determined by a function 
defined by Equations 13a, 13b, 13c and 13d below, for example. Such polygons are 
intended to be approximations of the concentric rings of Equation 10. The polygons of 
Equations 13a, 13b, 13c and 13d are twelve-sided. However, the embodiments are not 
restricted to twelve-sided polygons. The greater the number of sides to a given 
polygon, the closer the approximation to a ring. Thus, the level of approximation to a 
ring can be user-selected. 

Further, the V correction value may either be increased in decreased by a 
luminance parameter that is related to the Y value of the target pixel. Such a 
luminance parameter is user-selected based on whether the Y value of the target pixel 
satisfies certain luminance-related conditions. The luminance-related conditions 
associated with the V correction value may be defined by Equations 9a, 9b, and 9c 
below, for example. 

Thus, based on the above description, the V correction value, namely, DeltaV, 
may be defined by Equation 6 below, as an example, according to certain 
embodiments. 

According to certain embodiments, the Y correction value is based on is based 
on a second distance value. Such a Y correction value is bounded by a minimum and 
a maximum Y correction limit, which are constants. The minimum and maximum Y 
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correction limits may be user-selected based on properties of the lens. The user may 
select such minimum and maximum Y correction limits so that the resulting Y correction 
value is neither insufficient nor excessive when applied to a given target pixel. 

With respect to the second distance value, the second distance value is based 
on the first distance value (as previously described above), which in turn is augmented 
by user-selected constants. Such user-selected constants are based on the F value of 
the lens that is being corrected. The second distance value may be defined by 
Equation 14 below, as an example, according to certain embodiments. 

Further, the Y correction value is based on a smoothing parameter, referred to 
herein as an alpha function. For purposes of explanation, assume that the Y correction 
can be represented by a curve that has a discontinuous point at the lower limit of the Y 
correction curve (see Equation 4 below). The alpha function is such that it has the 
effect of smoothing out the discontinuous point at the lower limit of the Y correction 
curve. The smoothing parameter may be defined by Equation 7 below, as an example, 
according to certain embodiments. C1 is the location associated with the discontinuous 
point at the lower limit of the Y correction curve. C3, C4 and C5 are selected by the 
user and depend upon the degree of smoothing that is desired. 
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Thus, based on the above description, the Y correction value, namely, YGain, 
may be defined by Equation 4 below, as an example, according to certain 
embodiments. 

The following variables and functions are defined as follows: 

x : horizontal coordinate of a pixel 
y : vertical coordinate of a pixel 

Y : luminance value of a pixel 

U : U value of a pixel, (i.e., chrominance U = Blue - Y) 

V : V value of a pixel, (i.e., chrominance V = Red - Y) 

YGain = Function_YGain(x,y) 
DeltaU = Function_DeltaU(x,y,U) 
DeltaV = Function_DeltaV(x,y,V) 

where: 

YGain = Function_YGain(x,y) 

= min(YMAXLIMIT, max(YM IN LIMIT, Function_DistanceJwo(x,y) * 



DeltaU = Function_DeltaU(x,y,Y) 

= min(UMAXLIMIT, max(UM IN LIMIT, Function_Distance(x,y) * UCOEFF 



YCOEFF + YOFFSET)) + Alpha 



Equation 4 



+ UOFFSET))/uGain(Y) 



Equation 5 
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DeltaV = Function_DeltaV(x,y,Y) 

= min(VMAXLIMIT, max(VMINLIMIT, Function_Distance(x,y) * VCOEFF 
+ VOFFSET))/vGain(Y) Equation 6 

Alpha: 

assume Ytemp_gain = abs(min(YMAXLIMIT, max(YMINLIMIT, 
Function_Distance(x,y) * YCOEFF + YOFFSET)) 
if(abs(Ytemp_gain -d ) < c2) 

Alpha = (c3-abs(Ytemp_gain - c4))/c5; Equation 7 



uGain(Y): 

if (Y > c6) 
else if (Y < c8) 
else 



uGain(Y) = c7 
uGain(Y) = c9 
uGain(Y) = c16 



Equation 8a 
Equation 8b 
Equation 8c 



vGain(Y): 

if (Y > c10) vGain(Y) = d 1 Equation 9a 
else if (Y < d 2) vGain(Y) = d 3 Equation 9b 
else vGain(Y) = c17 Equation 9c 

where: 

d, c2, c3, c4, c5, c6, c7, c8, c9, c10, c11, c12, c13, c16, c17 are constants. 
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YMINLIMIT : constant value 
YMAXLIMIT : constant value 
UMINLIMIT : constant value 
UMAXLIMIT : constant value 
VMINLIMIT : constant value 
VMAXLIMIT : constant value 
YOFFSET : offset value 
UOFFSET : offset value 
VOFFSET : offset value 
YCOEFF: coefficient value 
UCOEFF: coefficient value 
VCOEFF: coefficient value 

Note that the specific values for the constants and coefficients above depend 
upon the specific lens/image sensor combination. In general, these constants and 
coefficients are obtained through experimental observations of a reference image 
obtained from the lens/image sensor combination and can be selected by the user 
based on such observations. 

FIG. 4 is a diagram illustrating a set of coordinates used in the correction 
method according to certain embodiments of the invention. FIG. 4 shows a reference 
image 402 with length 404 (XSIZE) in the X direction, height 406 (YSIZE) in the Y 
direction. Reference image 402 is associated with a coordinate system having an 
origin [0,0] at location 410. The center of reference image 402 is at location 412 with 
coordinates [XSIZE/2, YSIZE/2]. The location of pixel 408 may be described with 
reference to location 412. The location of pixel 408 may also be described with 
reference to an arbitrary location such as location 418 with coordinates [XSIZE/2 + 
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XSHIFT, YSIZE/2 + YSHIFT]. XSHIFT and YSHIFT represent the distances from 
location 412 in the X and Y direction, respectively. 
Let 

HalfX = XSIZE/2 

HalfY = YSIZE/2 

Dx= abs(HalfX-x + XSHIFT) 

Dy = abs(HalfY - y + YSHIFT) 

NormalizeValue is a value that depends on whether the implementation is 
an 8 bit or 16 bit etc., implementation. For example, for an 8 bit implementation, the 
NormalizeValue is a value which make the range of the final result of each of the 
equations 10, 11, 12, 13a, 13b, 13c, 1 3d from 0 ~ 255, wherein 2 8 -1 =255 

The Function_Distance(x,y) can be one of the following equations or other 
equivalent Distance function. 

Function_Distance(x,y) = Root( Dx*Dx + Dy*Dy) * NormalizeValue Equation 10 
or 

Function_Distance(x,y) = (Dx + Dy) * NormalizeValue Equation 1 1 

or 

Function_Distance(x,y) = max (Dx.Dy) * NormalizeValue Equation 12 
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or 

If (Dx > (Dy«2)) 

then, Function_Distance(x,y) = ( Dx + Dy +max(Dx , Dy) 

+ (abs(Dx -(Dy«2))»3)) ) * NormalizeValue Equation 13a 

Elself(Dy> (Dx«2) ) 

then, Function_Distance(x,y) = ( Dx + Dy +max(Dx , Dy) 

+ (abs(Dy -(Dx«2))»3) ) * NormalizeValue Equation 1 3b 

Else If (max(Dx.Dy) > (abs(Dx-Dy)«2)) 

then, Function_Distance(x,y) 

= (Dx + Dy +max(Dx , Dy) + (max(Dx,Dy) - (abs(Dx- Dy)«2)»3) ) * 
NormalizeValue Equation 13c 

Else 

Function_Distance(x,y) 

= (Dx + Dy +max(Dx , Dy) ) * NormalizeValue Equation 13d 

where, "«2" means multiply 4, and "»3" means divide by 8. 
The various equations set forth above are alternative embodiments that have 
various implementation advantages/disadvantages. For example, Equation 10 is not as 
easy to implement in hardware (cost will be high) and it require more calculation 
resources in software. However, the resultant image for Equation 10 is relatively good. 
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Equation 11 is easy to implement in hardware (cost will be low) and it requires 
relatively small computational resources in software. However, the resultant image is 
not as good as that of Equation 10. 

Equation 12 is easy to implement in hardware (cost will be low) and it requires 
relatively small computational resources in software. However, the resultant image is 
not as good as that of Equation 10. 

Finally, Equations 13a, 13b, 13c, 13d are easy to implement in hardware (cost 
will be low) and it requires low amounts of computational resources in software. 
Further, the resultant image is good. 

Assume: 

Distance = Function_Distance(x,y) 
Function_Distance_two(x,y) = 

Distance - max(0,(Y -(c14-Distance/c15))) Equation 14 

where: 

c14, c15 are constants and can be selected by the user based on the F value of 
the lens for which lens correction is to be applied. 

Max(a, b): Taking a bigger value between a and b 
Min(a, b): Taking a smaller value between a and b 
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